---
id: llamaindex
title: LlamaIndex
sidebar_label: LlamaIndex
---


import Tabs from "@theme/Tabs";
import TabItem from "@theme/TabItem";
import { Timeline, TimelineItem } from "@site/src/components/Timeline";
import VideoDisplayer from "@site/src/components/VideoDisplayer";

# LlamaIndex

[LlamaIndex](https://www.llamaindex.ai/) is an orchestration framework that simplifies data ingestion, indexing, and querying, allowing developers to integrate private and public data into LLM applications for retrieval-augmented generation and knowledge augmentation.

:::tip
We recommend logging in to [Confident AI](https://app.confident-ai.com) to view your LlamaIndex evaluation traces.

```bash
deepeval login
```

:::

## End-to-End Evals

`deepeval` allows you to evaluate LlamaIndex applications end-to-end in **under a minute**.

<Timeline>

<TimelineItem title="Configure LlamaIndex">

Setup tracing for LlamaIndex and create an Agent. Use `trace` context manager to set up the `AgentSpanContext` (or `LlmSpanContext` if you want to evaluate the LLM span).

```python title="main.py" showLineNumbers
import asyncio

from llama_index.llms.openai import OpenAI
from llama_index.core.agent import FunctionAgent
import llama_index.core.instrumentation as instrument

from deepeval.integrations.llama_index import instrument_llama_index
from deepeval.tracing.trace_context import AgentSpanContext
from deepeval.metrics import AnswerRelevancyMetric
from deepeval.tracing import trace

instrument_llama_index(instrument.get_dispatcher())


def multiply(a: float, b: float) -> float:
    """Useful for multiplying two numbers."""
    return a * b

agent = FunctionAgent(
    tools=[multiply],
    llm=OpenAI(model="gpt-4o-mini"),
    system_prompt="You are a helpful assistant that can perform calculations.",
)

answer_relevancy_metric = AnswerRelevancyMetric()

async def llm_app(input: str):
    agent_span_context = AgentSpanContext(
        metrics=[answer_relevancy_metric],
    )
    with trace(agent_span_context=agent_span_context):
        return await agent.run(input)
```

:::info
Only metrics with LLM parameters `input` and `output` are eligible for evaluation.
:::

</TimelineItem>
<TimelineItem title="Run evaluations">

Create an `EvaluationDataset` and invoke your LlamaIndex application for each golden within the `evals_iterator()` loop to run end-to-end evaluations.

<Tabs groupId="llamaindex">
<TabItem value="asynchronous" label="Asynchronous">

```python title="main.py" showLineNumbers
from deepeval.dataset import EvaluationDataset, Golden

dataset = EvaluationDataset(
    goldens=[Golden(input="What is 3 * 12?"), Golden(input="What is 4 * 13?")]
)

for golden in dataset.evals_iterator():
    task = asyncio.create_task(llm_app(golden.input))
    dataset.evaluate(task)
```

</TabItem>
</Tabs>

✅ Done. The `evals_iterator` will automatically generate a test run with individual evaluation traces for each golden.

</TimelineItem>
<TimelineItem title="View on Confident AI (optional)">

<VideoDisplayer
  src="https://confident-bucket.s3.us-east-1.amazonaws.com/end-to-end%3Allama-index-1080.mp4"
/>

</TimelineItem>

</Timeline>

:::note
If you need to evaluate individual components of your LlamaIndex application, [set up tracing](/docs/evaluation-llm-tracing) instead.
:::

## Evals in Production

To run online evaluations in production, simply replace `metric_collection` in `AgentSpanContext` with a [metric collection](https://documentation.confident-ai.com/docs/llm-tracing/evaluations#online-evaluations) string from Confident AI, and push your LlamaIndex agent to production.

```python filename="main.py" showLineNumbers
async def llm_app(input: str):
    agent_span_context = AgentSpanContext(
        metric_collection="test_collection_1",
    )
    with trace(agent_span_context=agent_span_context):
        return await agent.run(input)
```
